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Foreword 

In May of 1984, NASA initiated an Automation and RoboticsTechnology Planning Study in 
response to a mandate from the U S. Congress associated with appropriations for the 
Space Station. 

Boeing Aerospace Company initiated this study of the Operator-System Interface (OSI) in 
August of 1984 in response to a need expressed by NASA for study coverage in that area 
The study: 


• Characterizes an OSI for an extra vehicular (EV) robot system to perform 
maintenance functions on the Space Station, 

• Develops OSI senarios for that system, and 

• Assesses the associated technologies 

Dr Victor Ansel mo was the NASA manager of the study and the Boeing effort was lead by 
Paul Meyer of Boeing Aerospace Company, who was supported by Dr. Douglas Dorrough 
and Ron Hammond of the Boeing Computer Services Artificial Intelligence Center. Other 
contributers to this report are Joe Hopkins, Henry Lahore, Mark Lawler, Judi Qualy- 
White, and AmyToussaint. 
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EV Robot 
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1.0 INTRODUCTION 

This is the final report of a Space Station Automation and Robotics Planning Study, which 
was a joint project of the Boeing Aerospace Company, Boeing Commercial Airplane 
Company, and Boeing Computer Services Company. Figure 1-1 shows the work 
breakdown for the Boeing study tasks. This study is in support of the Advanced 
Technology Advisory Committee established by NASA in accordance with a mandate by 
the U S Congress. Our support complements that provided to the NASA-Contractor study 
team by four aerospace contractors, the Stanford Research Institute (SRI), and the 
California Space Institute. This study identifies automation and robotics (A&R) 
technologies that can be advanced by requirements levied by the Space Station program. 
The methodology used in the study is to establish functional requirements for the 
operator-system-interface (OSI), establish the technologies needed to meet these 
requirements, and to forecast the availability of these technologies 

Boeing entered the study in the third month of a six month effort to address the OSI issues 
(sometimes called man-machine interface issues) The other aerospace companies 
working on this study focused on functional aspects of automation and robotics including 
subsystem management, space manufacturing, free-flyer servicing and space construction, 
but none of the contractors were specifically tasked to address the OSI The OSI is integral 
to the other topics and affects Space Station technology growth considerations as human 
involvement in Space Station caretaking is replaced by automation and robots The OSI 
topic chosen for this study is not controls and displays, which are relatively well 
understood, but rather the advanced automation functions that define these interfaces. 

The roll of SRI in the NASA-Contractor group was to provide focused technology forecasts 
to support the analysis and to guide the system concept design performed by the 
aerospace contractors. Because contracted tasks were set before Boeing joined the study, 
the technology support provided by SRI was not available for our part of the study. The 
Boeing Computer Services (BCS) Artificial Intelligence (Al) Center provided similar support 
for the OSI topics we addressed. The BCS Al Center is particularly well suited to perform 
the required technology definition and forecasting tasks because of their connection with 
studies on similar topics that have been done for other users Figure 1-2 shows a 
functional organization chart for the BCS Al Center. 
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Figure 1-1: Work Breakdown Structure 
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1.1 Scope of Study 

The overall OSI for the Space Station covers operator interfaces to a wide range of 
automation and robotics functions, including subsystem management, planning, mission 
management, maintenance management, logistics management, free-flyer servicing and 
operation, and proximity operations. Each of these involves the display of monitoring, 
diagnostic, and advisory information to the crew and the acceptance of planning or 
discrete command inputs from the crew The software functions required to generate, 
interpret, and manage the information, as well as to perform the decision making needed 
for diagnostic and advisory outputs, will lead to technology advancements The system 
characteristics and senarios that describe those software functions constitute the concept 
definition output of our study. The technology identifications and forecasts related to 
those functions are the outputs of our study which support technology planning. 

Study schedule and resource limitations required us to focus on a specific aspect of OSI. 
We selected a topic that drives out advanced software technologies but is credible for 
Space Station use within 10 to 15 years after the initial operational capability (IOC). As 
shown in Figure 1-3, our study looked at progressively more detailed Space Station 
functions, starting from general stationkeeping functions, down to proximity operations, 
and finally to the extra vehicular (EV) robot functions. The EV robot we envision would be 
a free-flyer while in transit from one location to another in close proximity to the orbiting 
Space Station The OSI would perform path planning, tracking and control, object 
recognition, fault detection and correction, and plan modifications in connection with EV 
robot operations. The implementation of the OSI implies the use of natural languages, 
voice recognition and synthesis, speech understanding, expert diagnostic and advisory 
knowledge systems, and machine learning The technologies for these implementations 
are expected to evolve through three distinct phases, as discussed in Section 4 Figure 1-4 
shows a flow diagram indicating how software development could support OSI for an E\ 
robot 
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1.2 DEFINITIONS 

The following established definitions have been used in this report and are included here 
for easy reference. 

Automation 

Automation is the use of machines to effect control of system/subsystem processes in a 
predefined or modelled set of circumstances 

Artificial Intelligence 

Al is the part of computer science concerned with the design and implementation of 
programs that make complicated decisions, learn or become more adept at making 
decisions, interact with humans in a way natural to humans, and in general exhibit the 
characteristics we associate with intelligent human behavior Intelligence, as used here, is 
the ability to meet and cope with novel situations by adjusting behavior, the ability to 
comprehend the interrelationships between facts and concepts, and the ability to 
generate new concepts and relationships from those already known, i e., already in a 
database Artificial, as used here, indicates that intelligence is achieved by means of a 
computer or electro-mechamcal-optical device. 

Autonomy 

Autonomy is an attribute of a system/subsystem that will allow it to operate within its 
specified performance requirements as an independent unit or element without external 
intervention for a specific period of time. 

Expert System 

Expert or knowledge based systems are systems that use a significant amount of expert 
information about a particular domain to solve problems in that domain. The system is 
able to perform at the level of a human expert in that domain of knowledge. 
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Knowledge Engineering 

This discipline involves wih extracting, articulating, and computerizing knowledge. 
Knowledge engineering addresses the problem of building skilled computer systems by 
extracting the expert's knowledge and then organizing it in an effective implementation. 

Machine Autonomy 

Machine autonomy is defined as the ability to function as an independent unit over an 
extended period of time, while performing a variety of actions and while responding to 
stimuli produced by integrally contained sensors 

Robot 

A generic term, connoting many of the following ideas A machine capable of 
manipulating objects and/or movement having enough internal control, sensing and 
computer analysis to carry out a more or less sophisticated task. The term usually connotes 
a certain degree of autonomy and an ability to react appropriately to changing conditions 
in its environment 

Teleoperation 

A teleoperated robotic system is one that utilizes cybernetic anthropomorphic machine 
systems (CAMS) technology in order to permit the human operator to transmit his or her 
intelligence and dexterity through the machine and to the task All decision-making 
capability resides with the human controller A servo-control system usually transmits a 
small proportion of the load force to the operator's hand(s), thus giving him or her 
"instinctive control" of the job. Frequently, six degrees of freedom are present These 
include horizontal extension, hoist, azimuth rotation, yaw, pitch, and roll. 

1.3 Organization of this Report 

This report presents an overview of the study, describes an OSI concept for EV robot 
operations based on a hypothetical task scenario and astronaut/system interactive 
dialogueue, makes a technology forecast, and sets forth conclusions and 
recommendations. 
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2.0 STUDY OVERVIEW 

During the second session of the 98th Congress, the appropriations for Space Station 
funding were established by a House/Senate conference In their report, the Senators and 
Representatives on that committee emphasized automation and robotics as part of the 
Space Station Program, as the following quote from that report illustrates* 

The Space Station Program offers an opportunity to stimulate 
the development of advanced technologies in the fields of 
automation and robotics. To this end, the conferees adopted 
the Senate provision establishing an Advanced Technology 
Advisory Committee mandated to identify specific Space Station 
systems which advance those technologies that are not in use in 
existing spacecraft. Examples of such technologies include 
advanced vision sensors, computers that can serve as expert 
systems, and manipulator systems with advanced multiple 
degrees of freedom. The conferees intend that, where 
appropriate, the Committee may as a secondary task also 
identify systems currently in use whose potential for enhancing 
automation and robotics technologies appears promising The 
conferees both intend and expect that the technologies of Space 
Station automation and robotics will be identified and 
developed not only to increase the efficiency of the station itself 
but also to enhance the Nation's technical and scientific base 
leading to more productive industries here on earth. 

In response to this directive, NASA established an Aerospace Contractor Study Group to 
cover four specific areas of automation and robotics application to the Space Station. 

Satellite Servicing - TRW 

Space Manufacturing - General Electric 

Space Assembly and Construction - Martin Marietta 

Subsystems Management - Hughes 

Initially Boeing was not a participant, but offered to assist the NASA effort by studying 
the impact of the operator-systems interface on Space Station automation. Boeing has 
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significant experience in that area as part of the advanced commercial airliner flight deck 
development. The following is a list of some significant features of the OSI approach to 
757/767 flightdeckdesign 

• Integrated digital instrumentation 

• Flight deck commonality 

• Simplified procedures 

• Increased automation/decreased workload 

• Consistent caution/warning philosophy 

• Optimized crew size and accommodations 

• Advanced human engineering design methodology 

Extensive pilot and customer participation 
Work load assessment 
Extensive engineering simulation 

• Quiet, dark cockpit 

OSI is the system of hardware and software that facilitates communication between 
human operators and the hardware/software system that monitors and controls a 
functional system On the Space Station, the functional systems that will be controlled and 
monitored will include those involved with housekeeping, stationkeeping, and mission 
and operations planning and scheduling. The following lists these functions and some of 
theirsub-functions: 

• Housekeeping 

Subsystem management 
Inventory control 
Resource management 
Inspection and maintenance 

• Stationkeeping 

Orbital maintenance 

Space Station and free-flyer formation control 

Free flyers approach control 

Proximity operations (manipulators/EVA) 

Momentum management 

• Planning and Scheduling 

Tasks 
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Logistics 

Mission 

All of these could be supported by the OSI to provide the on-board astronauts with a 
display of status, control, and advisory information, as well as a means of giving directives 
to the functions. The OSI would include software that resides in the Space Station data 
management computers or in special-purpose processors associated with a particular 
function. The software would support OSI input/output functions such as speech 
recognition or multifunction display processing and it could support the diagnostics and 
simulation processing that feeds information to the displays. Space Station constraints on 
OSI are shown by Figure 2-1. 

The human factors aspects of OSI design lead to the general requirements listed below. 

• The OSI shall be "user-friendly" and to implementthatthe OSI shall: 

• Provide feedback to the operator 

• Provide appropriate level of detail 

• Allow different ways for operator interaction 

• The OSI shall be multifunctional to minimize power, weight, and volume and 
to reduce operator workload and error rate 

• Information integration shall be used to reduce workload and error rate 

• Commonality in format and operation shall be maintained. 

As stated in Section 1 .0, this study focused on one aspect of the OSI. The EV robot function 
was selected because it represents a forward-looking application of automation and 
robotics and because of these additional factors- 

• The function identifies across-the-board OSI technology needs 

• It can be implemented without risking Space Station schedule or cost goals 

• It is within the OSI area and does not duplicate the work of pre-existing 
contractors 

• The technology is generic to many potential Space Station applications 

• The technology is applicable to Earth-based applications and will increase 
U.S. technical competitiveness. 
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• VARIETY OF MISSIONS AND MODULES REQUIRES VERSATILITY 

• ZERO-G ENVIRONMENT 

• POSTURAL CHANGES 

• LINE - OF - SIGHT FALLS 25 DEGREES BELOW 
HORIZONTAL REFERENCE 

• HEIGHT INCREASE 

• NORMAL OPERATING POSITION IS NOT SEATED 

• POSSIBLE CHANGES IN QUALITY OF VISION 

• DIVERSE BACKGROUNDS OF CREW MEMBERS 

• CREW MEMBERS NOT HIGHLY TRAINED IN ALL AREAS 

• CREW MEMBERS ATTENTION MAY BE DIVIDED AMONG 
MULTIPLE TASKS 

Figure 2-1 SPACE STATION ASPECTS OF OSI 

The principle method used in this study to characterize the use of OSI for an EV robot is the 
scenario of a day's operations by the robot and the associated operator interactions. 
Section 3.2 present the robot task scenario and an astronaut/system dialogueue that 
illustrates a specific OSI interaction with an astronaut. 

The EV robot system is envisioned to be a free-flying vehicle that will operate outside the 
Space Station. The robot will be equipped with manipulator arms to hold itself to a work 
site and to perform physical tasks at a work site. The vehicle would be plugged into a 
specific berthing port on the outside of the station while being programmed and 
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recharged with expendables It would be deployed from that port and travel by its own 
propulsion system near the Space Station to perform its assigned tasks. The primary 
advantage of the EV robot system is that it would increase crew productivity by reducing 
the amount of time required for EVA on routine and frequently-ocurring tasks and by 
performing tasks that exceed human capability It would also reduce risks to the crew by 
performing hazardous functions. Figure 2-2 depicts one robot design concept and figure 
2-3 illustrates a simple sequence of tasks that could be executed by an EV robot. 
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Figure 2-3 EV ROBOT TASK FLOW 


This report focuses on the OSI supervisory mode of directing a robot The scenarios in 
Section 3.2 were selected to drive out many generally applicable Al technologies Some of 
the technologies indicated by the scenarios and dialogueue of Section 3.2 include - 

voice recognition 
speech understanding 
natural language understanding 
machine reasoning 
image understanding 


2-7 









D483 10027-1 


adaptive data base management 
expert systems 
learning systems 

The scenario given in Section 3.2 1 indicates the range of tasks that an EV robot could 
perform to support Space Station maintenance, experiment, and astronaut EVA 
operations. The dialogueue in Section 3 2 2 describes an interactive OSI session during 
which an astronaut provides directions for completing a robotic task. This task, which 
modifies a similar task, requires the addition of some procedures developed during an 
experimental program. 

This instruction session is conducted by the astronaut in a work station in the Space 
Station. The astronaut is interacting with a software program within the Space Station 
computer system. This instruction program and simulation will be of a fidelity that, once 
the task is demonstrated to be understood, the instructions can be stored, appropriately 
assembled with other tasks, and downloaded to an EV robot prior to the time the task is to 
be performed 

When the robot leaves its berthing area it will have a schedule of tasks to perform, a 
travel path calculated to reach the task sites; a 3-D map of the static Space Station; and 
the intelligence to perform some deviations from the preprogrammed plan For example, 
it will have sensors and communication means which will keep it informed of its location 
and other objects moving in the proximity of the Space Station. The robot will be 
provided with collision avoidance procedures which will permit some plan deviation and 
still maintain autonomous operation. As the EV robot performs its tasks it will be able to 
report to and receive commands from the astronauts and the Space Station computers 

One of the most significant considerations in defining this OSI concept is the use of 
astronaut time. The OSI must accept high-level, verbal and graphics instuctions that can 
be input by the astronaut quickly and simply. Another consideration is the variation that 
would be inherent in each astronaut's delivery of high-level directions The OSI would 
need to have a natural language capability adaptable to all of the anticipated human 
users. 

Section 4.0 of this report gives a technology assessment that was performed by the BCS Al 
center. The assessment presents an evolutionary sequence for the development of A&R 
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technologies from the present to about 2010 The concepts described in Section 3.2 would 
fall into the Phase 3 of the assessment, as discussed in Section 4.2.3. 

The question of teleoperation versus autonomous robot evolution has been raised and is 
discussed in some detail by Section 4.4. 

The conclusions resulting from this study and a recommendation for an OSI advanced 
development program are included in Section 5.0. 
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3.0 DESCRIPTION OF THE OSI FOR AN EV ROBOT 

This Section describes the tasks that would be performed by a mature EV robot system; 
tasks that represent a significant pull on Al and robotic technology. These tasks are 
illustrated by two scenarios, one describing a full day's work by an EV robot and the other 
describing a task planning dialogue between an astronaut and the OSI. These scenarios 
are followed by a description of some of the operational requirements for an EV robot/OSI 
system, including OSI communication functions, task planning and scheduling, and 
anomaly management 

3.1 Critical and Routine Tasks 

The reasons for using robots on the Space Station are to relieve the crew of time- 
consuming, potentially hazardous, and highly repetitive tasks The critical tasks to be 
performed by an EV robot, such as handling hazardous materials, performing extended 
EVA operations and assisting with superhuman precision adjustments, are the design 
drivers exerting the most technology pull on the OSI system 

In addition to critical tasks, the EV robots can be expected to perform tasks that are day-to- 
day, predictable, well-defined, and repetitive housekeeping chores. These tasks, which 
include inspecting the Space Station exterior for damage or wear and removing 
contamination from exterior surfaces, do not represent an optimal use of crew time when 
performed through EVA. The unproductive "overhead" time required to suit up, gather 
materials, travel to and from the task site, and unsuit after the task is done, may well 
overwhelm the time required for the task itself. 

Another set of routine tasks well within the postulated capability of EV robots is 
experiment support. Many instruments used in space experiments will require routine 
servicing such as replenishing consumables, replacing focal plane instruments, changing 
film cannisters or optical filters, and placing or retrieving material samples While similar 
in required capability to the housekeeping tasks, these tasks are not as basic to EV robot 
services because they are not as routine. That is, the task requirements will change from 
experiment to experiment and the planning and robot programming for the task will 
probably have to be done on-station. Therefore, the savings in crew time are not as great 
as for automating housekeeping functions. These tasks will also depend on the existence 
of Space-Station-deployable, task-oriented planning software for the EV robots. 
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In addition to performing critical and routine tasks, EV robots may also be able to serve as 
crew assistants. A mature EV robot could be used as an assistant to a human crew member 
in addition to performing tasks autonomously. These capabilities could reduce the 
frequency or duration of EVA or reduce the number of crew members needed for some 
EVA tasks. One of the simpler crew support applications, which may be possible with a 
rudimentary EV robot system, is to use a robot to provide a remote view of a potential EVA 
site. The Space Shuttle has used a TV camera mounted on the remote manipulator for a 
similar application A robot carrying a TV camera or other sensor could be dispatched to 
provide crew visibilty of a remote site to aid in EVA planning The robot could also 
continue surveillance during EVA as a sensor for the crew member(s) monitoring the EVA. 
In a more sophisticated assistant application, the EV robot could act as a caddy, tagging 
after the EVA crew member, carrying and fetching tools and materials. The highest level 
of assistant task is for the EV robot to act as an extra set of hands in positioning tools and 
materials and to lend strength or precision to the crew member. 

The OSI between the EV robot and Space Station crew will evolve as robotics tasks become 
more complex. The OSI features that will be needed frequently, such as the 
communication functions discussed in the next Section, must be at the crew's fingertips 
and not just at a centralized command and control station. If a crew member must travel 
to a central console to issue frequent commands to a robot performing routine tasks, the 
time-savings of using the robot may be lost. Therefore, a portable remote 
communications system will be required. As the robot technology becomes more 
autonomous, the frequency of communication will probably decrease. The third level of 
tasks described above will require a natural language interface to on-line task planning 
and control. Such capabilities will certainly not be available when the EV robot system is 
used initially and will provide a significant technology pull on Al and robotics. 

3.2 Scenarios 

The following two scenarios illustrate how a robot might perform some of the types of 
Space Station-related tasks described previously. The first scenario summarizes a range of 
tasks that could be performed during one day, and the second scenario illustrates an OSI 
sequence between a crew member and the task planner for a relatively autonomous 
robot. These scenarios were created to establish an innovative idea envelope within which 
the conceptual designs could be developed and to which the forecasts for technologies 
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necessary to support an EV robot OSI development could be addressed. The events are 
extrapolated from what could be possible, based on the current state of the art, by about 
2000 to 2010. 


3.2.1 Full Day Scenario 

This scenario is presented to indicate the range of tasks that an EV robot might perform 
for the Space Station It provides the setting for our OSI concept definition. 

One morning after a busy night of tasks, the task planner (computer) creates a schedule of 
tasks for the robot, which are derived from the following requirements list 

• A sensory experiment requires a bad card changeout - URGENT 

• An observation experiment requires a film pack change. 

• Two experiments require battery pack changeout 

• Two experiments require routine calibration 

• One experiment has mounting problems - needs examination - astronauts 
had requested further examination from previous day 

• One astronaut requests 1.5 hours of robot time to assist in EVA task from 
0930-1100 

• Internal preventive maintenance period must be conducted. 


The schedule created leads to the following events 


0700 -- Robot performs routine self-check tests (replaces one joint servo that isn't 
within tolerance - logs change) 

0730 - -- Robot loads up with propulsion pack, 1 film cannister, 2 battery packs, 1 
computer card, calibration test equipment 


0750.-- 


0810. -- 
0825. -- 


Robot moves toward first task (begins sensor observation of exterior while in 
transit). During observation notes 3 configuration differences - places 
message for astronaut review 
Arrives at sensor experiment - replaces bad card 

Selects path to next task Begins movement toward next task (starts sensor 
observation of exterior). Notes 2 anomalies Expert system determines 4 of 
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the 5 anomalies found so far are near the damaged sensor which was just 
repaired Initiates low-level annunciation signal for astronaut alert. 

0836- - - Arrives at experiment - changes battery pack. 

0844. - - Selects path to next task. Moves toward next package (because of proximity 
to current location, no exterior observation). 

0850: -- Arrives at experiment. Earth observation camera E16B - changes film. 

0915: - - Robot moves to airlock to load up with tools required for EVA assistance task, 
places exposed film in airlock for astronaut attention, and changes out 
propulsion/powerpack. 

0930. - - Robot arrives with astronaut at EVA operations site 
Another robot joins task to perform 'gofer' functions 

0950: - - Robot provides spotlight illumination for astronaut as Sun sets 

1025: -- Robot senses sunrise and turns off spotlight. 

1 100. - - EVA task ends. Astronaut interrupts schedule - asks robot to return to area of 
damaged sensor (repaired earlier). Robot performs detailed inspection under 
direction of two astronauts It appears that a micrometeroite may have 
caused damage. Astronauts decide to perform EVA at 1300 to further 
examine area and select any necessary repair options. The robot's presence is 
requested, which causes the task planner to update the schedule. 

1145.-- Robot selects path to nearest task (turns on observation sensors while 
traversing - no discrepancies noted). 

1153.-- Arrives at experiment - performs calibrations. 

1217: - - Selects path to next task - observes exterior while traveling. 

1226 -- Changes batteries on experiment 

1229- -- Selects path - warned of OMV approach in proximity of task - waits for 
completion of OMVdocking 

1247 -- Moves off to next task - observes exterior 

1 308. - - Arrives late at EVA inspection site Had been monitoring astronaut progress, 
but calibration had taken longer than planned, and astronaut had arrived 
earlier than projected - error noted for further scheduling considerations. 

1 344 • - - Astronauts decide repair will be performed later in week after more time has 
been given to assessing and preparing repair options 

At this point, there are no other immediate tasks for the robot to perform or assist on. 

However, the task planner has a number of functions stored that are either standard 
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servicing requirements that need to be integrated into the robot's schedule or are new 
requests that were issued when the robot was occupied but that didn't require an 
interruption because there was no specified time. The following is a list of these 
outstanding requests, which is followed by a task scenario that shows how they are 
integrated into the next segment of the robot's schedule. 

New requests 

• Assist a crew member in positioning an experiment package. MRMS will also 
be used. Robotto provide additional eyepomt 

• Clean exterior surface of Window C within the next 36 hours to accommodate 
an Earth-observation photo session from station interior. Location of smudge 
on window noted. 


Unscheduled maintenance/support tasks 


• Perform preventative maintenance on 2 experiments based on trend failure 
analysis 

• Repair a piece of hardware on the solar arrays based on analysis indicating 
excessive degradation 

• Complete an instrument calibration based on projections of a busy schedule 
when the normal calibration would occur. 

• Survey sector 3D-2 sub 6. This is required because no recent activity has 
occurred in this vicinity. 


1345:-- Robot moves to home lock area 

1353: -- Unloads equipment used on prior tasks and switches out power packs Loads 
up with maintenance tools. 

1403: -- Moves to first experiment site. 

1412: - - Begins maintenance (cleaning, adjustment, replacement). 

1435 -- Moves to 'early' instrument calibration via sector 3D-2 sub 6. Performs 
exterior survey during traverse. 

1457: -- Performs calibration of instrument package 
1 503: - - Receives message that astronaut-assist task will begin at 1 530 
1509: -- Moves to site of astronaut-assist task. 
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1518. -- Arrives at site early Notifies crew of arrival, goes into background mode 
while awaiting task start. 

1530- -- Provides mobile "flying" eyepoint for MRMS maneuvering. This mobility is 
directed by natural language/communication techniques. 

1610- -- Robot switches on spotlightto provide illumination. 

1623: - - Task completed. Returns to home lock area for window cleaning equipment. 

1637: -- Unloads equipment, picks up cleaning tools. 

1648:-- Moves to window. 

1701:-- Cleans window Astronaut happens to be in area and sees task Interactively 
suggests additional spots on window that need attention. 

1743:-- Moves to next task 

1754:-- Performs routine preventative maintenance task. 

1827:-- Returns to home lock. 

184V-- Unloads equipment Selects tools for solar array repair. 

1852 1 - - Moves to spares storage lock area. 

1908 -- Collects array parts that will be replaced. 

1923.-- Moves to solar array area requiring attention. Reduces speed when near 
array to reduce danger of collision and minimize contamination; continually 
senses array location as it rotates 

2003.-- Begins repair of structure 

2121 -- Moves to Scientific Airlock #2L (Large) 

2139 -- Unloads degraded struts and mirrors (for later analysis by crew) 

2153:-- Moves to home lock 

2204 -- Plugs self in. Nothing urgent on schedule, performs internal checks 

2217--- Task-planner schedules next series of tasks from low-priority lists. Notes 
restriction that crew will be bedding down for night shortly and that the 
habitat module will be off-limits for outside maintenance. 


3.2.2 Operator/System Dialogueue 

The first scenario showed the types of EVA a highly developed robot could perform to 
support the Space Station crew. Based on that kind of support function, the following 
script represents an OSI "dialogueue" between the EV robot system and astronaut that 
shows how the OSI instructions are conveyed 
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Let's examine in some detail how the robot task planner is directed to program a change 
to the task listed at 0850 in the first scenerio. The astronauts are installing a new camera 
to perform Earth-sensing photography. Before the camera becomes operational, a 
member of the crew must conduct a dialogue with the software that develops task plans 
for the robot. This software, which will be contained in the Space Station's computing and 
data management system, would have access to a CAD database describing the station 
and its equipment It would also have access to existing robot task plans and to simulation 
and graphics software. 


The result of the dialogue between the crew member and task planning software would 
be a new set of instructions for an EV robot. These instructions would have been 
developed and verified cooperatively by software and the astronaut and demonstrated by 
simulation. A crew memberwould probably observe the first execution of the new task to 
further verify the task plan. The dialogue would be along these lines: 


Astronaut 

Audio 

Astronaut 

Computer Display 
Astronaut 

Astronaut 

Audio 

Astronaut 

Computer Display 

Astronaut 

Audio 

Astronaut 

Audio 


"Hello, Task Planner." 

"Task Planner here." 

"We need a new film cannister exchange program at this location; 
computer graphics please " 

(CAD solids model version of Space Station is brought up ) 

Rotates and frames display to show robot's berth and the location 
of the work site 

"Note the location" -- points to the work site location with 
pointing device. 

"OK." 

"The cannister is for the E16C camera, which is like the old E16B; 
computer comparison graphics please." 

(Color coded CAD solids models of the E16C and E16B cameras are 
overlaid). 

"Note that the door is shaped differently please;" rotates the 

display simultaneously 

"OK." 

"The latch mechanism operation is different, note please," points 
to latch mechanisms with pointing device. 

"Show me." 
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Astronaut 


Computer 

Audio 

Astronaut 

Audio 

Astronaut 

Audio 

Astronaut 

Audio 

Astronaut 

Audio 

Astronaut 

Computers 

Astronaut 

Audio 


"The new latch is the same as the experimental version we used on 
test 0178-V3 computer, run simulation with graphics please, Task 
Planner, note please " 

(Runs simulation of the operation of the latch mechanism and 
displays the sequence). 

"OK." 

"The work site area is the same as before." 

"OK." 

"The photography schedule is the same as the G33A we ran for the 
E16B " 

"Start date 7 " 

"Day after tomorrow " 

"OK." 

"Notify me when each exposed film cannister is ready please " 
"OK." 

"Play that back please " 

(Runs simulation of task displays travel, task; elements, and 
schedule ) 

"Looks OK, store please " 

"Task Planner, OK and out." 
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3.3 OPERATOR - SYSTEMS INTERFACE 

The dialogueue presented in Section 3 2 2 describes an OSI that uses several advanced 
technologies. This Section discusses each of those technologies from a functional point of 
view. The OSI system that is implied in the discussion includes an internal robotic 
computer and embedded software, a centralized Space Station computer (task planner or 
scheduler), and related, supporting functions of the onboard Space Station data 
management system 

3.3.1 Communication Functions 

The operational philosophy for the OSI will be to provide a natural communication link 
between astronauts and the onboard computer system including the robot. For the 
astronauts, natural communication means input of data and commands using methods 
such as speaking and pointing, and receiving feedback through visual and aural channels 
To the computer and its associated peripherals, natural communication implies the 
necessary hardware and software required to accept from and express data to the 
astronauts in a lucid manner, indeed, the hardware and software should be transparent 
(i.e , unambiguous) to the user The astronaut should not be required to learn a new 
language or rigid command syntax rules. 

The dialogue given in Section 3.2 2 represents an OSI concept that uses this natural 
communication philosophy. The conceptual design described below delineates the 
components and operational requirements of each OSI element However, the description 
does not include component layout, OSI geometry, or hardware specifications The 
components envisioned for this OSI include color graphic displays, voice recognition and 
synthesis, and nonverbal communication links (keyboard, hand controller, and touch 
inputdevice). 

3. 3.1.1 Displays 

Pictorial and graphics displays will be the primary method for presenting visual data to 
Space Station crew members, although some control operations may require direct 
viewing These displays must integrate data into an easily comprehendible format to help 
the operator understand and act on the data presented Studies have shown that 90% of 
the information we process is received through visual data, most of which is perceived as 
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objects, not words or numbers. Therefore, the displays will present mostly dynamic or 
static pictorial graphics presented in real time, which will be supplemented by 
alphanumeric data. The displays may be presented on a large flat screen (as opposed to a 
CRT) at a central control station and on smaller portable screens that will be part of the 
remote, possibly hand-held OSI communication link. 

There are two OSI display concepts -- head-up (HUD) and holographic displays -- that 
would not be required for the dialogueue presented in Section 3.2 2 but should be studied 
in more detail. A head-up display (HUD) is an instrument that projects computer- 
generated dynamic symbology onto a clear combining surface mounted in the operator's 
field of view (FOV), thereby overlaying the symbology on the viewed scene. The operator 
then can have all necessary information in the immediate FOV, decreasing eye 
accommodation and attention diversion problems. A head-up display could be used for 
both IV and EV activities. 

A holographic display presents a true three-dimensional representation of an object or 
situation to the viewer This type of display would be useful for flight operations, to 
simulate a repair task, as a teaching/learning aid, and to evaluate construction techniques. 
This technology requires some pushing and direction for this application. The specific 
areas of concern in using holographic displays include power consumption, processing 
time, and real-time processing techniques. 

3. 3. 1.2 Voice Recognition and Synthesis 

The voice recognition system will be used to send commands to the computer and robot. 
It will provide the astronaut with a natural means of communication - the spoken word. 
The same commands could also be entered through a keyboard or touch input device. To 
truly meet the natural communication requirement, the recognition system will have to 
accommodate a flexible syntax for messages from the crew, although conventional rules 
of grammar will be observed. The system must also recognize continuous speech, allowing 
recognizable words or phrases to be spoken at a natural rate The system will be designed 
to recognize a speaker by requesting the speaker to recite selected phrases which are 
recorded by the system. The recordings will then be analyzed for speaker-unique 
pronunciation of the phonemes included in the selected phrases and these phonemes will 
be compared to the standard phoneme database whenever the speaker issues a command. 
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The comparison will allow the system to interpret the speaker's command. This technique 
minimizes the expenditure of crew time for both system training and use. 

The recognition system must also be speaker-independent and speaker-adaptive. 
Speaker-independent means that the system will recognize a large number of speakers 
without losing accuracy However, a speaker will be required to identify him or herself for 
recognition and security purposes when first using the system Speaker-adaptive means 
that the system is flexible enough to recognize a speaker during various stress levels of the 
speaker's voice. 

As the dialogueue showed, the astronaut gams entry into the task planning mode for the 
robot by simply saying, "Hello, Task Planner." This indicates that the OSI recognizes the 
astronaut's voice and determines that the astronaut is a valid person to give task direction 
to the robot. The astronaut speaks conversationally and, in fact, doesn't use the most 
quantitative speech possible: he says "day after tomorrow" for the start date, rather than 
a specific date The OSI system correctly interprets the input, employing what it "knows" 
about that particular astronaut's use of the language so that the "day after tomorrow" is 
in Earth calendar days, corresponding to the astronaut's sleep-wake cycle. 

A voice synthesis system is a natural communication means between the computer and 
robot, and the crew Using voice synthesis as a feedback to the crews offloads their 
already overloaded visual channel and increases the usage of the traditionally 
underloaded aural channel. To allow the greatest flexibility, the synthesis system will also 
be phoneme-based. The system will create words by using the phonemes in the database 
In addition, the voice-type, (i.e., male/female, accent) will be selectable by the crew 

The OSI system synthesises audio responses which may, as shown in the dialogueue, 
initially be somewhat structured. As the system progresses, the audio responses would 
preferably be unstructured and, in fact, be intentionally changed to indicate real 
understanding of the directions received. 

3. 3. 1.3 Data Display and Exchange 

The astronaut and robot/OSI system simultaneously evaluate CAD data The computer 
system, on voice command from the astronaut, brings up the designated CAD data for 
display. In reaction to inputs from the astronaut through advanced input devices (i e., 
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light pens, display pressure point application, etc.), the computer manipulates the CAD 
data The OSI "observes" the data in real time and reacts to the astronaut's "Note the 
location", as the astronaut indicates the displayed location with a pointing device. 

To show comparisons graphically, the computer presents data for color-keyed displays 
with overlays and the system is able to interpret the distinctions in real time The OSI is 
also able to distinguish, in near real time, where things are the same or different by 
reviewing the data presented as the astronaut observes and manipulates the graphic 
display 

The computer system runs a simulation of sequences selected by voice input from the 
astronaut and simultaneously displays the results graphically The OSI system receives the 
data from the simulation as it is run and displays and interprets that data to identify 
features needed for the robot's task description. The OSI determines if the data is 
complete and consistent with the rest of the task description for an "OK" conclusion about 
the input 

3.3.2 ROBOT TASK PLANNING AND SCHEDULING 

The tasks that will be performed by the robot on orbit will be sequenced, integrated, and 
prioritized by the computerized task planner, in conjunction with commands issued by the 
crew through the OSI For example as we indicated in the first scenario, the mature robot 
will be able to function fairly autonomously in response to a preset schedule of tasks, in 
addition to being able to integrate unscheduled, routine tasks and emergencies This task 
planning capability will be preceded by simulation to assimilate the time, logistics, and 
procedural elements of the subtasks, which will be modified and updated as they are 
performed on orbit. The task planner, or scheduler, will receive inputs continuously from 
the crew and other computers, which will impose new requirements and constraints on 
the existing schedule Tasks that require immediate action will interrupt the existing 
sequence of events Each task input will have to take into account the completion time 
needed, paths of travel between tasks, and the resources that will be required to fulfill a 
task or schedule sequence (tools, power, fuel, parts). 

The task planner will also have to integrate information on other objects (Orbiter, OMV, 
OTV or free-flyers) maneuvering in the proximity of the Space Station or task area, in 
addition to any environmental constraints such as radiation emissions, signal blockages, 
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potential contamination sources (such as a thruster firing), and possible crew sleep 
disturbances. 

The robot will perform it's scheduled list of tasks either autonomously, checking for 
updates and changes after completing each task unless interrupted, or it may be involved 
in a direct astronaut/robot cooperative function, responding to immediate requirements 
as discussed in Section 3.1. 

The popular definition of intelligence includes the ability of an entity to learn from 
experience and avoid making any drastic mistakes. It is doubtful that a Space Station 
robot will be given full autonomy until it has such "intelligence " Confidence in the 
robot's "knowledge" will be gained by simulating many situations, including those in 
which an operator purposefully makes errors. A safety and sanity (S/S) monitor program 
should be developed concurrently with other robotic programs. This S/S monitor must 
have the ability to extract rules to cover similar future situations either directly by 
observation or, more likely, by means of interviews and conversations with operators 

3.3.3 ANOMALY MANAGEMENT 

While the robot may have increasing autonomy as crew confidence mounts and 
technology advances, there must still be a means for alerting the crew of faults and 
anomalous situations when they occur. An effective crew alerting philosophy should meet 
the following two general objectives - 

1. Minimize the time required for the crew to detect and assess failure 
conditions and to initiate correction actions, and 

2 Conform to the quiet, dark OSI concept when all systems are operating 
normally. 

Faults and anomalies should be categorized by the safety/sanity monitor into four major 
classes, each class eliciting a predefined alerting scheme. The first class is 
information/maintenance data, which include system trend data, maintenance log, etc 
and do not require immediate crew action or awareness No aural tone is elicited but a 
discrete indication (green/white) is given visually, (as shown in Section 3 2 1 at 0750). This 
data can also be recalled by the crew The next class is advisory data, which include 
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operational or system conditions, that require crew awareness and may require 
subsequent or future crew action. A prominent alphanumeric display (unique color) 
describing the advisory is provided, as well as a unique aural sound (see Section 3 2.1 at 
0825) The third class is caution data, which include abnormal operational or system 
conditions that require immediate crew awareness and subsequent corrective or 
compensatory crew action. A master visual (amber) indication, a prominent alphanumeric 
readout (amber), and a unique aural sound and voice message are issued. The last class is 
warning data, including emergency operational or system conditions, that require 
immediate corrective or compensatory action by the crew. This includes all time-critical 
faults A master visual (red) indication, a prominent alphanumeric readout (red), and a 
unique aural sound plus voice message are elicited These four classes are broad enough 
to encompass any forseeable fault or anomaly, and provide consistent nonconfiguring 
crew-alerting procedures 

As mentioned earlier, the safety/sanity monitor would categorize the faults. In addition, 
the monitor would prioritize detected faults, implement a safe and hold logic for critical 
situations, and broadcast a message for the fault to alert the system. The monitor reduces 
the number of crew alerts by solving minor faults and anomalies and thereby generally 
off-loads the crew and increases their confidence in autonomous system operations 

Another area of anomaly management involves situations during which a robot has 
difficulty with a prescribed procedure, or finds that a piece of hardware at a work site 
differs from that expected To some extent, a hierarchy similar to the alerting scheme can 
be followed in these situations For any situation involving a life-threatening or time- 
critical task the crew should be notified immediately Simultaneously, the robot system 
searches its own memory for a solution. If the situation is not life-threatening or time- 
critical, the robot should eventually be sophisticated enough to search for a solution 
autonomously and, only after exhausting potential solutions, notify the crew The 
solution search process would involve the safety/sanity monitor as well as the robot's 
memory 
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4.0 TECHNOLOGY ASSESSMENT 

Section 3.0 described the capabilities of the OSI that would supporta fully autonomous EV 
robot system This section begins by describing the technology assessment used to 
examine the feasibility of this OSI. Section 4.2 then uses the description of section 3 0 and 
results of the technology assessment to develop a three phase approach to EV robot 
autonomy. The technology advances required by each of these phases define the 
technology pull imposed by the Space Station EV robot OSI. Section 4 3 compares the 
technolgy pull defined in section 4 2 to estimates of technology availability to define the 
technology push that must be applied to develop the EV robot OSI. Section 4 4 discusses 
an alternative evolutionary path for the EV robot, that of building on current 
teleoperated robot technology, and concludes that a path based on partial autonomy is 
more likely to lead to a successful EV robot development. 

4.1 Technology Assessment: Project TAARGET 

Technology can be defined as a set of pragmatic principles, processes, and techniques 
derived by humans and intended for the manipulation and/or control of physical reality, 
including the reality of human-produced objects As defined, it is clear that technology is 
a pragmatic discipline or methodical effort, but not the results of such effort. Thus, 
structures, tools, machines, or any other artificial object is an artifact of technology, but 
not technology itself. However, artifacts can be used to estimate the status of the 
development of technology, thereby forming the basis for technology assessments 

A technology assessment (TA) is a set of statements and associated illustrations that 
describe the current and/or projected development status of a particular set of pragmatic 
principles, processes, and techniques by using the artifacts of same. A comprehensive TA 
requires an examination of current literature, consultation with experts in the technical 
area being assessed, and sophisticated forecasting and statistical methods to infer the 
future of the technology. Fortunately, a recent, full-scale assessment of robotic 
technology was available to support this study. Boeing used the Transnational Assessment 
of Autonomous Robotic Generational and Evolutionary Technology, (TAARGET), as a 
source of technology assessment for two reasons: because time constraints did not permit 
an independent, company-sponsored, effort; and the perceived excellence of the 
TAARGET effort itself. 
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Project target is described in Section 4.1 2 

4.1.1 Factors Influencing Technology Development -- Push and Pull 

The direction of a given technology is largely controlled by technology pull; that is, by 
operational and/or systems requirements the fullfillment of which necessitate the 
generation/development of relevant technical disciplines, products and processes. 

Technology pull starts with a set of operational requirements that serve as a necessary, but 
not a sufficient, condition for developing new technology These requirements are usually 
generated by the "customer" (e g , government agency, industy, organization, individual, 
etc ) and are usually a set of generic operations that a system will be expected to perform 
and against which it will be ultimately tested. Each one of these operational requirements 
must be translated into a detailed set of functions to be performed by any specific system 
expected to meet the operational requirements These functions are known as functional 
requirements They are independent of specific technologies in that more than one basic 
technology could be made to perform the functions dictated by the set (e.g., discrete vs 
integrated circuits). 

The next step translates functional requirements into a set of generic technology 
requirements (GTR) These may or may not be specific-technology independent They are 
"clusters" of interrelated technologies that can be used to perform major subsystem 
functions (e g , the inertial measurement unit or the guidance computer in a spacecraft or 
ballistic missile) 

The final step is very technology specific Where possible, it identifies specific techniques 
and processes that are mature enough for immediate support of the previously identified 
GTR's, functional, and operational requirements It may also identify techniques or 
processes that are not mature enough to meet these requirements. In that case, the 
requirements are said to "pull" the as-yet immature technology 

Obviously, technology pull is a polar concept that must be considered in relation to its 
opposite, technology "push", those relevent technical and/or socieconomic events that 
affect the arrival (maturing) time of a new technique or process. Socieconomic events are 
usually characterized by. 
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1) Generation dynamics: the elapsed time between one generation of a particular 
technology and the next 

2) Technology transfer time: the elapsed time between one major area of 
application and the next 

Both can be accelerated or decelerated by political and/or economic events. 

Predictive technology assessment tries to determine whether the push will equal the pull 
within a given lapse of time. As with TAARGET, the most credible assessments perform 
this determination quantitatively. 

4.1.2 Project TAARGET 

Some TA's are descriptive and/or cursory, others are sophisticated, quantitative statements 
regarding a particular technology. The TA for this study is derived from TAARGET, a large, 
methodologically quantitative, study of intelligent robotic technology This study was a 
three year, $12 million, international investigation that used the latest, most refined 
techniques currently available for a technology assessment The original TAARGET report, 
as well as it's 1983 update, are under priviledged title. 

Data Sources 

The TAARGET data sources are relevant documents as well as interviews with qualified 
non-Soviet-Block opinionators around the world. Twenty-two thousand documents 
related to research, development, and application of intelligent robotic systems and 
subsystems were examined and analyzed by people technically trained in the material 
being examined. Documents/papers in French, German, Italian, Japanese, Swedish, and 
English were examined to determine their implications for technology development. The 
analyzed material was used to structure an interview program to confirm/disavow the 
hypotheses generated by the document analysis. 

The principal tool used in the interview program was DELPHI - 14. This instrument 
permitted covert interviewing while generating reliable worst, best, and most likely time 
and performance estimates for subsequent input to the Decision Impact Risk Evaluation 
and Control Technique (DIRECT) simulator, which is discussed later. 
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One of the critical issues in developing a reliable consensus is the reliability of the 
opmionators consulted. A set of metrics were developed and tested on a random set of 
opinionators. These were found to be highly accurate in separating unreliable 
opimonators from reliable ones The metrics were applied to a population of 4600 
potential opmionators in the free world Of these, 2000 were selected for interview. The 
greatest care was exercized by the project TAARGET team to ensure the reliability of its 
data sources. 

Data Evaluation Methodology 

After appropriately reliable data was obtained, it was processed by the risk assessment 
simulator, DIRECT, that was developed and refined during the late 1960'sand early 1970's. 
As indicated in Figure 4-1, the input format is that of distributions taken from the DELPHI 
exercise. These are then piecewise convoluted and the terminal distribution is processed 
by a set of specially developed risk equations Some of the outputs from DIRECT are given 
in Table 4-1. A typical example of a preliminary output from DIRECT is given in Figure 4- 
2a. A related, derivative output is shown in figure 4-2b. Both of these outputs are 
parameters in a quantitative reliability technology risk assessment 
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RISK ELEMENT "A" RISK ELEMENT "B" TOTAL RISK 



Figure 4- 1 

DIRECT: MAIN PROGRAM 
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1) COST/SCHEDULE /PERFORMANCE RISKS 

2) RISK BUDGET FOR EACH ITEM OR SUBSYSTEM TO BE DEVELOPED / PRODUCED 

3) CONTRACTOR FEE ON ANY TYPE OF INCENTIVE CONTRACT 

4) RISK OF MAKING TARGET FEE /PROFIT ANY TIME DURING CONTRACT FULLFILLMENT 

5) TOTAL PROGRAM IMPACT OF RISK ITEMS 

6) COMPARATIVE EVALUATION OF RISK ABATEMENT STRATEGIES 

7) OPTIMAL ALLOCATION OF RESOURCES (MANPOWER, DOLLARS) 

8) RISK OF REACHING PROJECTED TECHNOLOGY GOALS 

9) RISK OF MAKING A TARGET RETURN ON INVESTMENT (ROI) 

10) PRESENT WORTH AND RISK OF PAY BACK PERIOD 

11) MERGER RISKS 

12) RISKS AND RISK IMPACT ASSOCIATED WITH CHANGES IN PERFORMANCE REQUIREMENTS 

Table 4-1 

PARTIAL OUTPUTS FROM DIRECT 



RESOURCES (COST, 
MANHOURS, ETC) 


INTERRELATION OF RISK PARAMETERS 
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PROB 

MEETING 

MTBF 



(MTBF) 

(MTBF) 

(MTBF) 


TIME 

(YRS) 

Figure 4 -2b 

TYPICAL DIRECT OUTPUT 


DIRECT uses these preliminary results to simulate a PERT (or GERT) network with which 
impact analyses are performed. These involve quantitatively identifying the impact of 
socio-economic events (technology push conditioners) upon the rate of development to be 
expected from a given technology under differing socio-economic conditions. The results 
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of these analyses are usually a set of charts displaying best, worst, and most likely years in 
which differing types of demonstrations of the particulartechnology are to occur. 

Most sophisticated predictive technology assessments use a set of well-defined event to 
define the maturity of a technology Demonstrations are palpable events in technology 
development so the TAARGET study used a set of demonstrations to predict technology 
status. The types of demonstrations of a particular technology for which DIRECT predicts 
future status's are given in Table 4-2, together with the defining characteristics of the 
demonstrations 


CONCEPT FEASIBILITY 

- Usually Stand Alone 
Usually Non -Real Time 

- Crude, Undeveloped Interface(s) 

No Form, Fit Optimization 

MATURE LABORATORY PROTOTYPE 

Pulled by Clearly Defined Technical Objectives 

- Frequently Embedded 
Usually Real -Time 

- Interfaces Defined and Developed 

- Can Function as Proto - Engineering Baseline 

- Specific Potential Applications Can Be Defined 

- Accurate Quantitative Development Risk Assessments Possible 

DEVELOPMENT ENGINEERING 


Evolving Design 

Constrained by Form, Fit and Economic Constraints 
Well-Defined, Well - Developed Interface(s) 

Real -Time 

Frequently Embedded 
Operational Parameters Testbed 


PREPRODUCTION PROTOTYPE 

- Direct Basis for FAB and Assembly Requirements 

- Direct Basis for Production Cost Estimates 

- Field Testable Against Contracted Operational Requirements 


Table 4-2 DEMONSTRATION TYPES AND CHARACTERISTICS 
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Summary of the OSI Technology Assessment 


TAARGET results, suggest that it will not be possible to deploy a fully autonomous EV 
robot and OSI by the time of Space Station IOC. Therefore, we have defined a three phase 
approach to development of the EV robot and OSI. These three phases are marked by the 
technology pull of three separable sets of operational requirements The system 
deployed at IOC will have sufficient autonomy to provide significant benefit to the Space 
Station crew. As time passes the system will evolve into the fully autonomous system 
described in Section 3.0. Table 4-3 characterizes the initial and final states of this system. 



1995 • 


2010 • 


Receive and understand an expanding set of question and/or 
command sequences. 

Report inability to execute a command sequence. 

Report valid reasons for inability to execute a command 
sequence. 

Report location. 

Report intended movement from one task space to another 
Report orientation in task space. 

Report status of own subsystems. 

Report fault-intollerant failures 

Receive, understand, and verify objectives menu 

Report plan of accomplishment (P. of A ). 

Receive and understand corrections to P. of A. 

Infer and report intended changes in menu sequencing, with 
reasons. 

Identify and describe emergency situations 


Table 4-3 

SCENARIO -DERIVED OSI OPERATIONAL REQUIREMENTS 
(STRESSING SCENARIO) 


Table 4-4a identifies the three phases of autonomy in our plan 
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PHASES (DEGREES) OF AUTONOMY 


Current 

Phase 1 


Phase 2 

Phase 3 

• Human Operator 

• Robot in Primary 

• 

Robot is both • 

Robot 

provides all 
control and 

control loop 


primary controller 
and planner 

provides 
own goals 

problem solving 

• Human is Planner 


functions 

• Robot carries out 

• 

Human provides • 
intermediate goals 

Human 

controls 

• Full 

plan in sequence 


size and 

Teleoperation 

prescribed by 

• 

Robot devises 

content 


human 


task to meet 

of goal 
suite 


• Human Disabler 

• 

Human Disabler 





• 

Human 

Disabler 


Table4-4a 

GENERIC EVOLUTION OF AUTONOMY 


A stepped decrease in human control and monitoring can be inferred from Table 4-4a,. 
though Table 4-4b shows this stepped decrease more explicitly and also shows a stepped 
increase in robot autonomy 


Most 

4 

4 Phase 1 (IOC) 

4 

4 

4 Phase 2 (2000) 

4 

4 

4 Phase 3 (2003-2007) 

4 

Least 


Human Directed, 

(Low Autonomy) 

Human Monitored 
(Moderate Autonomy) 

Human Instruction and Crisis Intervention 
(Maximum Autonomy) 


Table 4-4b 
DEGREES OF OSI 


These phases into which the technology pull divides OSI development are based upon 
degrees of robot autonomy These phases will, in turn, be translated into sets of 


4-10 



D483 10027-1 


functional requirements, each of which is both OSI - and Phase - specific Such translation 
is accomplished in the following Section. 

4.2.1 Phase 1: Human Directed 

If the E.V A. robot is expected to carry out a plan prescribed in detail by a human controller 
under conditions of intelligent monitoring by that controller it must meet, at least, the 
following functional requirements. 

* • Receive/understand limited set of voice question/command sequences 

* • Receive/understand bar code graphics also seen by human operator 

• Nonvocal copy-back to operator of question/command sequence in 
synonymous vocabulary 

• Nonvocal copy-back to operator of bar code graphics 

* • Infer current position and terminus coordinates for next transit 

• Report location and transit orientation by nonvocal transmission of 
"perceived" barcode/reflectors 

• Report orientation to task space by nonvocal transmission of perceived 
bar code and tactile pressure sensors 

• Report completion of each step in task sequence by barcode. 

Of the above requirements, the first four and the last three are OSI-specific. The key 
technologies supporting these requirementswere selected on the basis of their 
evolutionary capacity. These technologies are - 

* • Voice Intentionality Constrained Evaluator (VICE) 

• RF/Optical Dual Mode Communication Link 

* • Knowledge-Based 2-D Image Understanding/Semantic Processor 

* • Inference Engine 

* • Integration with Primary Controller 

These technologies were selected not only for their pullability but also because, if pushed, 
they would permit a "nonscarring" evolution with respect to the next phase. One 
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technology listed is not preceded by an asterisk. This is because it was considered by the 
TAARGET team to be technologically mature 

VICE is an "interim" nonscarring technology being developed at Boeing's Artificial 
Intelligence Research Center as well as at at least two other laboratories overseas. It 
consists of introducing a small number of synthetic particles into natural speech in order to 
disambiguate words or phases that are normally clarified by visually observed, situational 
context. Thus, if future speech understanding involves the functions found in Figure 4-3 
then such particles would be a part of the "situation pragmatics" or the error "correction 
rules". 
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Figure 4-3 

FUTURE SPEECH RECOGNITION 




SEMANTIC 

RULES 


SITUATION 

PRAGMATICS 


ERROR 

CORRECTION 

RULES 


*> 


SPEECH 

INPUT 


Knowledge-Based Image Understanding (KBIU) uses intra- and interframe (picture) 
semantic relationships as well as some of the techniques of intensional logic to understand 
the nature of a scene or series of scenes. While image understanding technology has been 
pushed by the DARPA initiative in that area, it is still in need of both technology pull and 
corresponding push. 

To do the logic required by the KBIU as well as making inferences about current and 
terminal positions, an "inference engine" that is modularly up-gradable is required. 
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Robot control of its primary loop (RCPL) presupposes technology pull and push that is 
related to but apart from that of OSI For a robot to carry out the step-by-step commands 
associated with Phase 1 OSI, it must be able to control its movements sufficiently to avoid 
obstacles and handle complex pressure-sensitive tasks To do these adequately, 
technology in the areas of T V camera-augmented Khatib control (TVKC) and LADAR 
proximity sensing must be pulled and correspondingly pushed. The technology push for 
both of these was addressed by the Project TAARGET team Their most-likely estimates are 
given in Figure 4-4. 


Figure 4-4 

RCPL SPECIFICATION OF TECHNOLOGY PUSH FOR OSI PHASE 1 


TOULOUSE STANFORD BOEING 


PROXIMITY 

LADAR 

(MULTIPLE 

SENSOR JPL FANUC GM/FANUC 

SYSTEM) 

ZZ 


m: zz 


76 77 78 89 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 
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4.2.2 Phase 2: Human Monitored 

If an EVA robot is expected to plan and carry out intermediate goals set by a human 
controller, then several machine intellience technologies must be pulled by these 
functional requirements: 

* • Receive/understand connected speech with expanding vocabulary 

* • 3-D acquisition of objectives in space (with barcode backup) 

* • Vocal copy-back in synonymous language of aM transmissions from operator 

to robot. 

* • Nonvocal copy-back to operator of all transmissions to robot. 

* • Plan optimal/sub-optimal sequences for fulfilling operator goals. 

* • Initiate and execute unplanned obstacle avoidance 

• Periodic recording of all robotic subsystems status for eventual copy-back 

• Robot communicates its maintenance requirements vocally and nonvocally 

* • Robot perceives and communicates a limited set of crisis conditions to 

operator. 

Nearly all of the items on the list are preceded by an asterisk, i.e , considered pullers Two 
that are not starred are actually derived from others on the list that are. However, only 
the first, third, and fourth items are directly OSI pullers. 

The technologies pulled by these three are given in the list below. 

Voice Recognition 

* Speech Understanding 

* Language Representation 

* Natural Language Understanding 

* Analogical Reasoning 
Monotonic Reasoning 

* NonmonotonicReasoning 

* Speech Synthesis/Translation 
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Again, these were selected not only for their pullability but also because, if pushed, they 
would permit a "nonscarring" evolution with respect to the third phase Those items 
preceded by an asterisk are pullable and, in most cases, in need of technology push from 
items such as economic conditioners. Of the items on the list, perhaps three need some 
description. Language Representation, Monotonic, and Nonmonotonic Reasoning. 

Like everything else perceived by humans, language must be properly symbolized in order 
to be manipulated for understanding and synthesis. This is particularly a problem for 
machine intelligence if the representation uses excessive storage and processing 
capability. Efficient schemes for symbolizing and manipulating strings of language "data" 
must be devised and tested against human criteria for performing the same manipulative 
exercise. 

"Monotonic" and "nonmonotonic" reasoning are synonymous respectively with deductive 
logic (usually the first order predicate calculus) and the logic of belief and "hunch" 
statements Both are required in order for a machine intelligent robot to represent and 
conceptually synthesize its replies to a human interrogation. 

It should be noted that the speech and language requirements placed upon the Phase 2 
robot in order to facilitate OSI neither negate nor render obsolete the software for voice 
recognition developed for Phase 1. Rather, it builds upon it, improving each of the 
function boxes in Figure 4-3. 

4.2.3 Phase 3: Human Instruction and Crisis Intervention 

An EV robot required to provide its own goals (within limits) and then devise strategies to 
meet these goals must not only build on the development of Phases 1 and 2 but also be 
capable of these additional functions. 

* • Learn New . . 

- Objects 

- Properties 

- Relations 

- Events 

- Situations (Internal/External) 

- Human Voice Prints/Imagery 
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* • Autonomously Change Its Databases 

- KnowWhen Deficient 

- Know Required Information 

- Know How to Reorganize File Structures 

- Know How to Merge/Sort 

* • Autonomously alter its primary and/or secondary control loops as 

functions of learning 

* • Repair/Maintain Itself. 

All of the requirements listed above are technology pullers. Essentially the Phase 3 robot 
would be a vehicular, multidemensional expert system. The robot's knowledge base and 
its manipulation would not only have depth and breadth, but would also possess a 
floatable modularity, permitting limited inductive changes in its software 

The OSI-specific puller is involved with learning. The abilities to adapt to new 
surroundings and to solve new problems are important characteristics of intelligent 
entitites. These can be subdivided into two equally important components; acquisition of 
new knowledge and problem solving both to integrate the new knowledge and to deduce 
new information in the absence of presented facts 

The functional requirements listed above can be translated into these very pullable 
technologies: 

* • Learning Systems 

* • Adaptive Database Management 

* • Self-Adaptive Control 

* • Floating Architectures 

* • Reflective Interpretation 

* • Self Repair 

These technologies almost correlate one-for-one with the functional requirements that 
pull them. Again the OSI specific technology is learning systems, since it is expected that 
much of the robot learning will occur by means of an astronaut teacher. 
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4.3 Phased OSI Technology Push 

DIRECT was used to produce worst, best and most likely estimates of the years in which the 
four types of demonstrations discussed earlier would occur for OSI-specific technologies. 
However, only the most likely estimates are presented in this report. The primary reason 
for this is the highly parametric nature of "best" and "worst" estimates. That is; such 
estimates are related to a set of complex, interrelated, technical, economic and social 
factors the values of which must be varied over a broad range and in several dimensions. 
TAARGET used both macro- and microeconomic models to handle the interrelationships 
and to arrive at a spectrum of worst case and best case estimates Presentation of these 
estimates requires many pages of explanation, tables and graphs Instead of presenting 
this voluminous material, the following relationships can be used as a rough estimate of 
the impact of economic changes on the occurance of demonstrations of OSI-related 
technology 

1. Influence of decreased funding 

If the funding, from all sources, supporting development of a very nonmature technology 
is reduced for 12 months then the influence on the date of occurance of technology 
demonstrations is- 

% reduction 
0 - 15 
16 - 22 
23 - 33 
34 - 50 

2. Influence of increased funding 

If the funding, from all sources, supporting development of a very nonmature technology 
is increased for 12 months then the influence on the date of occurance of technology 
demonstrations is 

% increase subtract from "most likely" date (months) 

0-15 8-12 

16 - 22 13 - 20 


add to "most likely" date 
12 - 18 
19 - 30 
31 - 41 
42 - 60 
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23 - 33 21 - 28 

34 - 50 29 - 36 

Most of the OSI technologies for which a "push" assignment was given fall in the 
"nonmature" or "very nonmature" categories defined by TAARGET. 

4.3.1 Phase 1 Push 

Figure 4-5 shows the technology push and estimated technology demonstration dates for 
four OSI-related technologies. Only the first two of these are needed for the Phase 1 OSI. 
Thethird, Inference Engine, will be required forthe Phase 2 OSI development. 

The project TAARGET team developed a set of metrics to quantitatively describe 
technology performance at the time of the statusing demonstrations. In effect, these 
metrics constitute the minimum performance of the system expected at the time of the 
preproduction prototype demonstration. Table 4-5 gives the metric for each of the Phase 
1 OSI supporting technologies 
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VICE 


Figure 4-5 

OSI PHASE 1 TECHNOLOGY PUSH 




KNOWLEDGE -BASED 
2 - D PROCESSOR 



INFERENCE ENGINE 





INTEGRATOR WITH 



PRIMARY LOOP 
CONTROLLER 

78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 
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Table 4-5 

METRICS FOR OSI PHASE 1 TECHNOLOGY PUSH 


VICE 


f • 2000 WORD VOCABULARY 

< • 1% ERROR RATE 

\ « PROCESSING SPEED = HUMAN 


KN. BASED 2 - D 
PROCESSOR 


4 FRAMES PER SECOND 
t • INTERFRAME ANALYSIS > HUMAN 
\ • <200 RULES 

V.® ACCURACY = HUMAN 


INFERRENCE 

ENGINE 


f • MONOTONIC REASONING 
J • 20 NON -CHAINED INFERENCES PER MINUTE 

\ • ACCURACY = HUMAN 


4.3.2 Phase 2 Push 


Figure 4-6 presents the push for the supporting technologies for the phase 2 OSI. The most 
likely estimates make it clear that preproduction prototypes for some of these will not be 
available until afterthe year 2000 This is especially true for analogical and nonmonotonic 
reasoning. 
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Figure 4 - 6 

OSI PHASE 2 TECHNOLOGY PUSH 


VOICE RECOGNITION 


A 


LIMITED SPEECH UNDERSTANDING 


A A 


LANGUAGE REPRESENTATION 


A 


NATURAL LANGUAGE UNDERSTANDING 


A 


ANALOGICAL REASONING 


MONOTONIC REASONING 


A A 


NONMONOTONIC REASONING 


SPEECH SYNTHESIS 


A A 





84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 


This figure suggests that appropriately funded research and exploratory development efforts 
should be begun as soon as possible and continued for several years in these areas. 

• language representation 

• nautral language understanding 
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• analogical reasoning 

• nonmonotonicreasoning 

This recommendation is based not only on the estimated slip into the next century of OSI- 
sigmficant demonstrations but on the fact that all four technologies are critical drivers of 
subsequent robotictechnology development. 

A set of performance statusing metrics for Phase 2 was developed as shown in Table 4-6. 
Like those of Phase 1, they were developed after interrogation of opmionators in real- 
time applications throughout the United States and in several foreign countries. 


Table 4 - 6 

METRICS FOR OSI PHASE 2 TECHNOLOGY PUSH 


SPEECH SYNTHESIS, SPEECH, 
LANGUAGE UNDERSTANDING 


5000 WORD VOCABULARY 
2% ERROR RATE 
PROCESSING SPEED = HUMAN 


ANALOGICAL REASONING 


NONMONOTONIC REASONING 


8% ERROR RATE = HUMAN 
~ 30 INFERENCES PER MINUTE 

19% ERROR RATE = HUMAN 
60 INFERENCES PER MINUTE 


MONOTONIC REASONING 


2% ERROR RATE = HUMAN 
20 INFERENCES PER MINUTE 


4.3.3 Phase 3 Push 

The strictly OSI-supporting technology for Phase 3 is that of learning systems. Figure 4-7 
shows that the "most likely" estimates for this technology will nto produce a real-time 
preproduction prototype until 2007. The reasons for this are many and complex. While 
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Samuels (1963 and forward), Newell (1960 and forward), Lenat (1977), Evans (1981) and 
Winston (1979) have all made progress in this area, it is still very poorly understood and 
therefore, poorly developed. 


Figure 4-7 

OSI PHASE 3 TECHNOLOGY PUSH FOR LEARNING SYSTEMS 



90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 07 


Partofthe problem involved development of other, not strictly OSI, technology during the 
Phase 1 and Phase 2 periods These are 

• sensor fusion 

• image understanding 

• stereo vision 

• information fusion 

• heuristic search 

• knowledgeacquisition 

® knowledge representation 

Figure 4-8 and Table 4-7 give the push assessments for these technologies along with the 
metrics used to measure their status 
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OSI PHASE 3 TECHNOLOGY PUSH 


HEURISTIC SEARCH 




KNOWLEDGE 

ACQUISITION 




KNOWLEDGE 

REPRESENTATION 




UNDERSTANDING 
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INFORMATION 

FUSION 



82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 00 


In summary, learning is a problem-solving activity that depends heavily upon the development of 
intelligent vision systems as well as upon heuristic search and knowledge representation. It is, 
therefore, obvious that investigations in each of these areas must be accelerated and broadened. 


\ 


/ 

/ 


/ 



D483 10027-1 


Table 4-7 

METRICS FOR OSI PHASE 3 TECHNOLOGY PUSH 


HEURISTIC SEARCH 


KNOWLEDGE ACQUISITION 



ACCURACY = HUMAN TEST SUBJECTS 
SPEED = HUMAN TEST SUBJECTS 
SEARCH COMPLEXITY = GAKY 
DYNAMIC UTILITIES (Good) 


HARTLEY -MITCHIE- GOOD METRIC 


KNOWLEDGE REPRESENTATION (NONE AVAILABLE) 

SENSOR FUSION POLITOPOULOS CRITERION 


IMAGE UNDERSTANDING 


DORROUGH/HOLBEN CRITERION 


STEREO VISION 


NONE 


INFORMATION 


NONE 


4.4 Alternative Paths to Robotic Autonomy 

The position taken in this study is that maximum likelihood of success in developing the EV 
robot and its OSI and that maximum applicability of the developed technology to other 
fields will result if the basic approach is to make use of evolving machine autonomy 
throughout the development process 

However, an alternative development path would be to evolve from current teleoperation 
technology. This Section examines that alternative. Before addressing some of the 
questions related to the "movement" from teleoperation to intelligent autonomy, a 
characterization of each seems required. 
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A teleoperated robotic system is one that utilizes cybernetic anthropomorphic machine 
systems (CAMS) technology in order to permit the human operator to transmit his or her 
intelligence and dexterity through the machine and to the task. All decision-making 
capability resides with the human controller A servo-control system usually transmits a 
small proportion of the load force to the operator's hand(s) thus giving him or her 
"instinctive control" of the job. Frequently, six degrees of freedom are present. These 
Include horizontal extension, hoist, azimuth rotation, yaw, pitch, and roll 

'Machine autonomy' is defined in the National Bureau of Standards Dictionary as "the 
ability to function as an independent unit over an extended period of time, while 
performing a variety of actions and while responding to stimuli produced by integrally 
contained sensors " 

As characterized, the two concepts are far apart. Neither implies the other. It is, 
therefore, logical to ask whether or not there is some causal or perhaps evolutionary 
relationship between the two. 

Pursuing this direction of thought leads to the following considerations 1 Human 
teleoperators have succeeded in using teleoperations to program a robot to carry out a 
series of procedures for executing a particular set of rather simple tasks (e g., Kinsey, et al, 
also, Yonemoto, Takeuchi, and Cornfield ) A few, more complex, systems have been 
demonstrated by NASA with respect to planetary landers By some, therefore, it is 
regarded as reasonable to believe that a completely specified set of determimtic 
procedures which are related to a predetermined spectrum of task spaces for a given robot 
can make that robot autonomous in the sense defined above. 

There are two major difficulties with this position 1) It requires either that the number 
and kind of tasks performed be low and simple or that the computational burden be 
prohibitively high; and more significantly 2) the technology pull from such a position lies 
within the range of zero to very low. 

An alternate position (and one with large technology pull) is to assume that autonomy 
involving a maximal number and complexity of tasks together with manageable 
computational burdens will only come about by improving the supporting technologies of 
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machine intelligence while reserving teleoperation for 1) the small number of intense 
crisis situations that may arise in the course of a mission, and 2) robot learning. 

This latter is the Boeing posision It is supported by the responses from qualified 
opimonators within research communities around the world. To these was posed the 
following question by members of Project TAARGET- 

"AS DEFINED, CAN AUTONOMOUS MACHINE (ROBOTIC) 

TECHNOLOGY EVOLVE NATURALLY (i.e., WITHOUT MAJOR 
TECHNICAL INNOVATIONS/CHANGES) FROM CURRENT 
TELEOPERATED ROBOTIC TECHNOLOGY?" 


Their responses, summarized in Table 4-8, make it clear that almost all individuals involved 
in robotic research or its supporting technologies agree that autonomous robotic 
technology development will occur only by means of inventions/innovations apart from 
teleoperation. 
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RESEARCH COMMUNITY* 

YES (%) 

NO (%) 

Artificial Intelligence Lab (MIT Cambridge) 

0 

100 

Robotics Institute (Carnegie- Mellon) 

0 

100 

Fraunhozer Institut (Karlsruhe) 

3 

97 

Institute fur Informatik (Bonn) 

5 

95 

Institute fur Informatik (Karlsruhe) 

2 

98 

Labztoire Automatique (Montpellier) 

0 

100 

Comp Science Dept , G M Research Labs (Warren) 

0 

100 

A 1 Center, SRI International (Menlo Park) 

2 

98 

Robot Research Laboratories (Kingston) 

0 

100 

School of Artificial Intelligence (Edinburgh) 

0 

100 

Robotics Section, Hitachi Central Research Laboratory (Tokyo) 

0 

100 

Robotics Laboratory, Institute of Tech (Tokyo) 

1 

99 

Electrotechnical Lab, Science and Tech Agency (Tokyo) 

0 

100 

Central Laboratory, Kawasaki Heavy Industries (Tokyo) 

0 

100 

Umv of Maryland Comp Sci Center (College Park) 

4 

96 

Stanford Artificial Intelligence Lab (Palo Alto) 

0 

100 


* Minimum Sample Size = 4 

Table 4 - 8 

OPINIONATOR RESPONSE TO FEASIBILITY OF AUTONOMOUS ROBOTS 
EVOLVING FROM TELEOPERATION 
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5.0 CONCLUSIONS AND RECOMMENDATIONS 

The following major conclusions can be drawn from the results of this study. It is 
technically feasible to develop an automated OSI by about 2010 to perform efficient 
supervisory management functions for an EV robot. The results of this study show that it is 
technically feasible to develop an initial, fairly rudimentory EV robot and OSI system by 
about mid 1990, and a sophisticated, efficient, and convenient system by about 2010. The 
initial OSI system would have a limited supervisory capability and would be largely 
experimental in nature. 

The artificial intelligence technologies that will need to be pushed to develop OSI 
capabilities are- 

Language representation, 

Natural language understanding. 

Analogical reasoning, and 
Nonmonotonic reasoning. 

This study also showed that an EV robot will have great potential for relieving astronauts 
of routine and hazardous tasks and for increasing the level of EV activity in support of the 
Space Station To achieve this, the OSI for an EV robot will facilitate high-level 
communication between the astronaut and the system. 

To develop the complex systems and interactions between humans, hardware and 
software that are needed for the EV robot OSI, we recommend that NASA initiate an 
advanced development program in this area. The program could include a concept 
definition phase, using requirements identification and trade studies The trades would 
include the partitioning between human and robotic activities, the partitioning of 
processing between the Space Station DMS and the onboard robot processor, the use of 
fixed versus portable or dispersed controls and displays, the teleoperation versus 
autonomous robot operation trades, programming language trades, RF versus onboard 
data storage trades, robot design trades that affect OSI, Space Station design trades that 
affect OSI, and trades on the rate of evolutionary progression of the robot-OSI systems. 
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The second phase could include development of candidate OSI hardware and software 
concepts To facilitate this phase, it is recommended that a simulation of EV robot body 
dynamics and manipulator actions be developed and coupled with models of candidate 
OSI software, and controls and display hardware. The simulation of the EV robot would 
provide graphic outputs through use of computer-generated imagery as well as 
quantitative measures of performance. Through use of this tool, modelled OSI systems 
could be evaluated by human operators and modified to obtain more and more 
satisfactory interaction. When a workable interaction is reached with these models, the 
design requirements for an OSI demonstrator could be extracted 

The third phase of the program could demonstrate and further develop interactions 
between an EV robot and human users For this phase it is recommended that a neutral 
buoyancy facility test bed be used. The robot for such a test bed could initially be an-off- 
the shelf aquatic robot modified to include necessary on-board processing. The candidate 
OSI software could be resident in laboratory computers and in processors associated with 
the tankside control station and robot berthing port The test site control station could 
serve as a test bed for further development of candidate controls and display hardware. 
Using this test bed facility, the hardware, software, and techniques for OSI supervision of a 
robot could be further developed and demonstrated Operations where the robot 
supports an EVA astronaut could be developed and demonstrated by an astronaut 
working in the tank with the robot which is supervised by control station inputs through 
the candidate OSI. 

Such a program requires detailed implementation planning to bring the necessary 
facilities, test articles, hardware, software and personnel together into a fruitful effort. 
Realistic goals and milestones need to be established as well as test evaluation criteria 
Early initiation of such a program by NASA would not only help develop the EV robot-OSI 
concept and the associated technology but would also develop confidence in the concept 
by the potential Space Station users. 
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